Automatic Acquisition of Language Model based on Head-Dependent Relation between Words

Language modeling is to associate a sequence of words with a priori probability, which is a key part of many natural language applications such as speech recognition and statistical machine translation. In this paper, we present a language modeling based on a kind of simple dependency grammar. The grammar consists of head-dependent relations between words and can be learned automatically from a raw corpus using the reestimation algorithm which is also introduced in this paper. Our experiments show that the proposed model performs better than n-gram models at 11% to 11.5~ reductions in test corpus entropy. 1 I n t r o d u c t i o n Language modeling is to associate a priori probability to a sentence. It is a key part of many natural language applications such as speech recognition and statistical machine translation. Previous works for language modeling can be broadly divided into two approaches; one is ngram-based and the other is grammar-based. N-gram model estimates the probability of a sentence as the product of the probability of each word in the sentence. It assumes that probability of the nth word is dependent on the previous n 1 words. The n-gram probabilities are estimated by simply counting the n-gram frequencies in a training corpus. In some cases, class (or part of speech) n-grams are used instead of word n-grams(Brown et al., 1992; Chang and Chen, 1996). N-gram model has been widely used so far, but it has always been clear that n-gram can not represent long distance dependencies. In contrast with n-gram model, grammarbased approach assigns syntactic structures to a sentence and computes the probability of the sentence using the probabilities of the structures. Long distance dependencies can be represented well by means of the structures. The approach usually makes use of phrase structure grammars such as probabilistic context-free grammar and recursive transition network(Lari and Young, 1991; Sneff, 1992; Chen, 1996). In the approach, however, a sentence which is not accepted by the grammar is assigned zero probability. Thus, the grammar must have broadcoverage so that any sentence will get non-zero probability. But acquisition of such a robust grammar has been known to be very difficult. Due to the difficulty, some works try to use an integrated model of grammar and n-gram compensating each other(McCandless, 1994; Meteer and Rohlicek, 1993). Given a robust grammar, grammar-based language modeling is expected to be more powerful and compact in model size than n-gram-based one. In this paper we present a language modeling based on a kind of simple dependency grammar. The grammar consists of head-dependent relations between words and can be learned automatically from a raw corpus using the reestimation algorithm which is also introduced in this paper. Based on the dependencies, a sentence is analyzed and assigned syntactic structures by which long distance dependences are represented. Because the model can be thought of as a linguistic bi-gram model, the smoothing functions of n-gram models can be applied to it. Thus, the model can be robust, adapt easily to new domains, and be effective. The paper is organized as follows. We introduce some definitions and notations for the dependency grammar and the reestimation algorithm in section 2, and explain the algorithm in section 3. In section 4, we show the experimental results for the suggested model compared to n-gram models. Finally, section 5 concludes this paper. 2 A S i m p l e D e p e n d e n c y G r a m m a r In this paper, we assume a kind of simple dependency grammar which describes a language